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PA’s  New  Research  Controls:  j 
roblems  Remain.  j~ 


esponding  to  past  criticism  of  its  research 
quality,  the  Environmental  Protection  Agency 
established  new  research  control  procedures 
in  1978.  GAO's  review  of  these  new  proce¬ 
dures  indicates  that  although  improvements 
have  been  made,  more  still  needs  to  be  done. 

GAO  recommends  that  EPA: 


-Require  that  research  strategies  be 
developed. 

-Establish  procedures  that  require  re¬ 
gulatory  offices  and  researchers  to 
agree  before  projects  are  started  that 
the  approach  and  timing  are  reason¬ 
able  to  meet  intended  needs. 


-Improve  its  information  systems  to 
establish  an  adequate  mechanism  for 
monitoring  projects. 

-Monitor  research  progress  against 
approved  plans  and  strategies. 


-Require  external  peer  review  of  all 
research  strategies  and  consistent  peer 
review  of  extramural  and  inhouse  re¬ 
search  proposes. 
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United  states  general  accounting  office 

WASHINGTON,  O.C.  20548 


COMMUNITY  AND  ECONOMIC 
DEVELOPMENT  DIVISION 


B-203649 


The  Honorable  Anne  M.  Gorsuch 
Administrator,  Environmental 
Protection  Agency 

Dear  Ms-  Gorsuch: 


In  response  to  past  external  and  internal  criticism,  the 
Environmental  Protection  Agency  has  taken  various  steps  to 
improve  its  research  program.  Our  review  of  these  new  pro¬ 
cedures  and  practices  indicates  that  although  progress  has 
been  made,  more  still  needs  to  be  done. 


Our  report  contains  recommendations  to  you  on  pages  20  ana 
29.  as  you  know,  section  236  of  the  Legislative  Reorganization 
Act  of  1970  requires  the  head  of  a  Federal  agency  to  suumit 
a  written  statement  on  actions  taxen  on  our  recommendations 
to  the  House  and  Senate  Committees  on  Government  Operations 
not  later  than  b0  days  after  the  date  of  the  report  and  to  the 
House  and  Senate  Committees  on  Appropriations  with  the  agency's 
first  request  for  appropriations  made  more  than  60  aays  after 
the  date  of  the  report. 

We  are  sending  copies  of  this  report  today  to  the  four  com¬ 
mittees  mentioned  above  and  interested  legislative  committees 
of  both  Houses;  the  Director,  Office  of  Management  ana  Budget; 
interested  Members  of  Congress;  and  other  parties. 

We  appreciate  the  cooperation  and  courtesy  of  EPA's  staff 
during  our  review. 


Sincerely  yours, 
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GENERAL  ACCOUNTING  OFFICE 
REPORT  TO  THE  ADMINISTRATOR, 

ENVIRONMENTAL  PROTECTION 
AGENCY 

External  and  internal  criticism  prompted  the 
Environmental  Protection  Agency  (EPA)  to  take  a 
closer  look  at  its  research  program.  Conse¬ 
quently,  in  1978,  EPA  started  actions  to  improve 
research  controls  and  acknowledged  that  its  pro¬ 
gram  lacked  the  following: 


—A  coherent  research  strategy  that  related 
program  objectives  and  priorities  to  research 
activities. 


— Effective  communications  and  decisionmaking 
mechanisms  between  researchers  and  programs. 


Accession  For 


— Adequately  responsive  research  results. 

j 

— An  adequate  program  of  long-term  and 
anticipatory  research,  c 

— Consistent  research  quality. 

GAO’s  review  of  EPA’s  new  procedures  and 
practices  indicates  that  although  progress 
has  been  made,  more  still  needs  to  be  done. 
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Because  of  the  importance  of  research  in 
EPA’s  regulatory  mission,  GAO  evaluated  the 
degree  to  which  EPA  took  action  to  implement 
its  new  research  controls  and  how  these  new 
procedures  resolved  previous  research  plan¬ 
ning  and  management  problems. 


NEW  RESEARCH  CONTROLS  ESTABLISHED 


To  enhance  communications  and  develop  a  decision¬ 
making  mechanism  between  researchers  and  program 
offices,  EPA  established  14  research  committees 
comprised  mostly  of  research  management  and  pro¬ 
gram  office  staff.  These  committees  are  primarily 
responsible  for  (1)  developing  multiyear  research 
strategies,  (2)  reviewing  research  plans,  (3)  par¬ 
ticipating  in  the  development  of  research  budget 
plans,  and  (4)  reviewing  the  quality  of  ongoing 
and  recently  completed  research.  (See  p.  3.) 
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Additionally,  EPA  established  a  mandatory  quality 
assurance  and  a  long-term  anticipatory  research 
program.  GAO  did  not  assess  these  two  programs 
because  neither  was  in  place  long  enough  to  gauge 
their  effectiveness. 

RESEARCH  PLANNING  IMPROVED: 

BUT  PROBLEMS  REMAIN 


Because  of  the  new  research  committees,  communica¬ 
tions  between  researchers  and  users  have  improved 
and  responsiveness  of  research  findings  to  program 
needs  is  better  than  in  the  past.  However,  in 
many  cases,  the  committees  nave  not  achieved 
agreement  between  researchers  and  the  program 
offices  concerning  strategies  and  research  to  be 
performed . 

Only  three  of  the  seven  committees  GAO  reviewed 
had  produced  finalized  and  approved  strategies; 
none  were  approved  in  time  for  the  fiscal  year 
1982  budget.  Draft  and  finalized  strategies  GAO 
reviewed  varied  greatly  in  scope,  detail/  and 
presentation  of  program  office  needs  and  planned 
research  activity.  (See  p.  8.) 

Relevance  to  program  office  needs  of  many  currently 
planned  research  projects  has  been  questioned  by 
four  of  the  seven  research  committees  GAO  reviewed, 
without  satisfactory  resolution.  Committees  have 
no  authority  to  require  solution  of  these  problems, 
and  researchers  have  little  incentive  to  otherwise 
resolve  them.  Unless  the  groups  formally  agree 
that  planned  research  will  reasonably  meet  EPA’s 
priority  regulatory  needs,  criticism  of  EPA's 
research  will  continue.  (See  p.  11.) 

MONITORING  RESEARCH  PROGRESS 
AND  PEER  REVIEW  OF  RESEARCH 
PROPOSALS  NEED  IMPROVEMENT 

Monitoring  the  progress  of  research  projects 
against  approved  plans  and  strategies  is  con¬ 
sidered  an  integral  step  in  assuring  useful 
research  results.  Research  committees  need  to  be 
kept  apprised  of  significant  modifications  in  the 
conduct  of  planned  research. 

GAO  found  that  none  of  the  research  committees  it 
reviewed  are  performing  this  monitoring  function. 

Of  the  seven  research  committees  GAO  reviewed, 
four  do  not  intend  to  track  projects.  Three  com¬ 
mittees  intend  to  monitor  projects  in  the  future 
using  existing  information  systems.  However, 
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GAO  found  these  existing  systems  inadequate 
because  they  are  either  inoperative  or  do 
not  contain  sufficient  data  to  effectively 
monitor  projects.  (See  p.  21.) 

EPA  has  been  criticized  in  the  past  because  of 
uneven  research  quality.  Congressional  hearings 
in  19/6  and  1977,  as  well  as  studies  completed 
by  the  National  Academy  of  Sciences  and  the 
Office  of  Technology  Assessment  found  this 
criticism  valid.  Assuring  quality  research  is  a 
difficult  task,  but  generally  accepted  standards 
include  peer  review  and  publication  of  research 
results  in  scientific  journals.  EPA  has  made 
significant  progress  regarding  peer  review  and 
publication  of  final  research  results,  but  more 
needs  to  be  done.  Five  of  the  seven  research 
strategies  GAO  reviewed  were  still  not  exter¬ 
nally  reviewed,  and  peer  review  of  extramural  and 
inhouse  research  proposals  varied  between  labora¬ 
tories.  For  example,  of  the  four  laboratories 
GAO  reviewed,  two  required  external  peer  review 
of  their  more  significant  inhouse  research  pro¬ 
posals  while  the  remaining  two  only  required 
internal  review.  (See  p.  25.) 

RECOMMENDATIONS 

GAO  recommends,  that  the  Administrator,  EPA: 

— Require  research  committees  to  develop 
approved  research  strategy  documents. 

— Establish  procedures  that  require  regulatory 
offices  and  researchers  to  agree  before  proj¬ 
ects  are  started  that  the  approach  and  timing 
are  reasonable  to  meet  intended  needs. 

— Improve  its  information  systems  to  establish 
an  adequate  mechanism  for  monitoring  projects. 

— Monitor  research  progress  against  approved 
plans  and  strategies. 

— Require  external  peer  review  of  all  research 
strategies  and  consistent  peer  review  of 
extramural  and  inhouse  research  proposals. 

EPA1 s  Office  of  Research  and  Development  offi¬ 
cials  agreed  with  most  of  GAO’s  conclusions  and 
recommendations.  However,  they  disagreed  with 
the  recommendation  that  EPA  establish  procedures 
that  require  researchers  and  users  to  agree  before 
work  is  started  that  the  approach  and  timing  of 
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projects  are  reasonable  to  meet  intended  needs. 

They  said  that  general  agreement  on  project  outputs 
already  exists  and  that  a  requirement  for  last 
minute  approval  of  the  approximately  1,800  new 
projects  each  year  is  tantamount  to  agreeing  to 
what  has  previously  been  agreed  upon.  According 
to  these  officials,  this  would  inevitably  result 
in  the  slowdown  of  research  programs,  as  well  as 
a  vast  increase  in  bureaucratic  papershuffling. 

GAO  disagrees.  GAO  found  that  agreement  is  not 
always  reached  concerning  research  project  outputs 
and  not  all  program  offices  are  satisfied  with  the 
researchers  degree  of  responsiveness  to  their 
expressed  needs.  Unless  agreement  is  reached  on 
the  conduct,  content,  and  timing  of  planned 
research,  GAO  believes  untimely  and  less  than  use¬ 
ful  research  results  will  continue. 

Concerning  GAO's  recommendations  that  EPA  monitor 
research  progress  against  approved  plans  and  stra¬ 
tegies,  EPA  officials  said  that  they  agree  moni¬ 
toring  is  necessary  and  that  research  committees 
should  be  encouraged  to  monitor  research  but  felt 
that  this  monitoring  should  consist  of  only  re¬ 
viewing  strategic  research  plans.  They  believe 
monitoring  of  implementation  plans  for  specific 
projects  should  be  the  research  manager's  respon¬ 
sibility.  These  officials  also  agreed  that  spe¬ 
cific  users  could  be  identified  on  EPA's  various 
project  monitoring  reports,  but  would  prefer  that 
the  various  progress  reports  be  distributed  to 
specific  users  only  on  a  request  basis. 

GAO  continues  to  believe  that  research  committees 
have  a  responsibility  to  monitor  research  progress 
and  suggest  corrections  for  significant  deviations 
not  only  against  approved  strategies  but  also 
against  implementation  plans.  GAO  believes  this 
is  an  integral  step  in  assuring  useful  research 
results.  Similarly,  GAO  believes  EPA  needs  to 
distribute  its  project  monitoring  reports  to  spe¬ 
cific  users  of  the  research  results  because  indivi¬ 
dual  users  may  not  always  be  aware  that  a  research 
project  is  underway  to  fulfill  his  or  her  needs. 
(See  pp.  20  and  29-30.) 
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CHAPTER  1 


INTRODUCTION 


The  Environmental  Protection  Agency  (EPA)  tries  to  abate 
and  control  pollution  systematically  by  integrating  a  variety 
of  research,  monitoring,  standard-setting,  and  enforcement 
activities.  The  primary  mission  of  EPA's  research  arm,  the 
Office  of  Research  and  Development  (ORD) ,  is  to  provide  the 
scientific  information  that  regulatory  offices  need  to  develop 
and  enforce  regulations.  ORD  is  headed  by  an  Assistant  Adminis¬ 
trator  and  organizationally  is  equal  to  EPA's  major  regulatory 
program  offices.  ORD  manages  more  than  2,500  ongoing  projects 
through  six  headquarters  offices,  two  field  administrative 
offices,  one  research  information  center,  and  15  laboratories 
located  through  the  country. 

Excluding  the  construction  grants  program  for  publicly  owned 
wastewater  treatment  facilities,  approximately  30  percent  of  EPA's 
annual  budget  is  earmarked  for  research  and  development.  For  fis¬ 
cal  year  1981,  ORD  was  appropriated  approximately  $253  million 
dollars  to  carry  out  its  mission.  Approximately  70  percent  of 
ORD's  appropriations  is  used  for  extramural  research  through 
contracts,  grants  and  agreements  primarily  with  universities, 
private  commercial  firms,  nonprofit  organizations,  State  and 
local  governments,  and  other  Federal  agencies. 

EPA  research  and  development  activities  can  be  grouped  into 
three  basic  categories: 

— Short-term  regulatory  -  responds  to  short-term  or  specific 
regulatory  needs  of  the  Congress  or  an  EPA  program  or 
regional  office. 

— Long-term  regulatory  -  addresses  longer  term  program 
or  regional  office  operational  requirements  but  does 
not  support  immediately  planned  regulatory  actions. 

— Exploratory  -  conducted  primarily  for  developing  funda¬ 
mental  knowledge  and  principles  to  use  in  solving  cur¬ 
rently  intractable  problems  or  identifying  or  understand¬ 
ing  future  environmental  problems  for  which  no  specific 
regulatory  activity  is  currently  contemplated. 

PAST  PROBLEMS  WITH 
EPA  RESEARCH 


In  his  confirmation  hearing  before  the  Senate  Committee 
on  Environment  and  Public  Works,  the  Deputy  Administrator,  EPA, 
testified  that 


"*  *  *the  need  for  research  results  that  are 
immediately  applicable  to  regulatory  and  enforce¬ 
ment  issues  has  led  in  many  instances  to  ill-con- 
cieved,  poorly  planned,  and  hurriedly  executed 
studies  which  have  resulted  in  products  of  inferior 
quality.  This  has  culminated  in  a  wasteful  process, 
both  in  terms  of  time  and  EPA  monies.  It  has  also 
been  costly  to  those  being  regulated  and  the  credi¬ 
bility  of  the  Agency  has  suffered  as  a  consequence." 

Congressional  hearings  in  1976  and  1977  also  indicated  that 
ORD  was  not  satisfactorily  identifying  and  fulfilling  the  Agency's 
needs  for  sound  scientific  information  to  support  environmental 
regulatory  decisions.  Studies  completed  by  the  National  Academy 
of  Sciencies  and  the  Office  of  Technology  Assessment  reinforced 
this  finding.  These  critics  of  EPA's  research  program  attributed 
its  lack  of  success  to 

— poor  communication  between  ORD  and  the  Agency's  program 
offices, 

— insufficient  attention  to  long-term  research,  and 

— lack  of  peer  (experts  inside  and  outside  of  EPA)  review 
of  research  plans  and  results. 

As  a  result  of  these  criticisms,  the  Congress  directed  EPA 
to  undertake  a  study  of  its  research  and  development  activities 
to  determine  the  causes  of  and  develop  solutions  for  the  problems 
facing  the  Agency. 

EPA's  NEW  RESEARCH  CONTROLS 


In  response  to  the  congressional  directives,  EPA  presented 
a  report  to  the  President  and  the  Congress  entitled  "The  Planning 
and  Management  of  Research  and  Development  Activities  Within  EPA" 
dated  June  30,  1978,  that  outlined  the  steps  EPA  would  take  to 
correct  its  research  problems.  The  report  pointed  out  that  EPA 
lacked  the  following  fundamental  components:  (1)  a  coherent 
research  strategy  that  related  program  objectives  and  priorities 
to  research  activities,  (2)  effective  communications  and  decision¬ 
making  mechanisms  between  researchers  and  program  offices,  (3)  an 
adequate  program  of  long-term  and  anticipatory  research,  (4)  ade¬ 
quately  responsive  research  results,  and  (5)  consistent  research 
quality.  The  report  proposed  the  following  corrective  actions: 

1)  Identify  distinct  research  planning  units  based  on  regula¬ 
tory  programs. 

2)  Establish  permanent  research  committees  for  each 
research  planning  unit  and  a  separate  research  oversight 
committee  composed  of  EPA's  top  management. 
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3)  Tie  several  planning  and  management  processes  to  Agency¬ 
wide  management  processes  with  decisions  focused  on  those 
connections . 

4)  Implement  a  research  incentive  system  designed  to  assure 
responsiveness  to  program  and  regional  research  needs. 

5)  Incorporate  peer  review  mechanisms  throughout  the  planning 
and  management  process  to  improve  research  quality. 

Research  committees 


In  an  attempt  to  enhance  communications,  EPA  initiated  a 
pilot  project  in  1978  to  examine  the  feasibility  of  planning 
research  and  development  programs  by  committees.  The  commit¬ 
tees  were  comprised  of  representatives  from  ORD,  program  offices, 
and  other  primary  organizational  elements.  Research  committees 
were  formed  in  five  areas — drinking  water,  industrial  wastewater, 
pesticides,  mobile  source  air  pollution,  and  particulate  air 
pollution.  Each  committee  was  cochaired  by  a  representative 
from  ORD  and  from  the  appropriate  corresponding  program  office. 
After  the  year-long  pilot  effort,  EPA  concluded  that  the  committee 
system  was  hot  only  feasible  but  that  it  had  great  potential  tor 
enhancing  ORD's  responsiveness  to  the  program  office's  need  for 
scientific  information. 

In  a  March  23,  1979,  General  Guidance  for  Research 
Committees,  ORD  integrated  the  committee  framework  into  the 
Agency's  mission  by  expanding  the  number  of  committees  from  b 
to  12.  The  particulate  air  pollution  committee  was  expanded 
to  encompass  gaseous  pollutants,  and  the  following  seven  research 
committees  were  established:  radiation,  oxidants,  hazardous 
air  pollutants,  municipal  wastewater  and  spill  prevention,  water 
quality,  solid  waste,  and  testing  and  assessment.  The  guidance 
strongly  encouraged  EPA's  Regional  Office,  Office  of  Enforce¬ 
ment,  and  Office  of  Planning  and  Management  to  designate  repre¬ 
sentatives  to  accompany  the  ORD  and  program  office  cochairman 
and  participants.  Each  committee  was  charged  with  five  broad 
tasks : 

— Review  the  current  program  to  familiarize  members  with 
the  status  of  ongoing  work. 

— Review  the  upcoming  year's  plans  and  formulate  appropriate 
recommendations  on  improving  those  plans. 

— Participate  in  developing  budget  year  plans. 

— Develop  a  multiyear  research  strategy  with  sufficient 
detail  for  use  as  a  primary  planning  document  in  subse¬ 
quent  years. 

— Review  the  quality  of  ongoing  and  recently  completed 
research . 
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Two  more  research  committees--eneryy  ana  hazardous  emergency 
response — were  added  in  subsequent  years,  bringing  the  total 
to  14. 

Research  quality 

As  part  of  an  attempt  to  offset  criticism  concerning  the 
quality  of  EPA  research,  the  Assistant  Administrator  for  Research 
and  Development  issued  a  Peer  Review  Guidance  on  January  2d,  19bu. 
The  guidance  called  for  each  laboratory  to  finalize  plans  for 
obtaining  peer  review  of  its  research  programs,  projects,  and 
research  results. 

These  plans  were  to  include  provisions  for 

--ORD's  senior  laboratory  management  to  encourage,  when 
appropriate,  the  publication  of  research  results  in 
professional  literature; 

— peer  review  to  be  obtained  for  research  results  not 
published  in  professional  literature; 

— significant  projects  (those  involving  an  expenditure 
above  a  predetermined  level)  to  be  reviewed  by  at  least 
three  non-EPA  peers; 

— projects  not  designated  as  significant  to  be  reviewed 
during  the  annual  laboratory  program  review  by  the 
cognizant  Deputy  Assistant  Administrator;  and 

— non-EPA  peers  to  be  included  whenever  possible  during 
the  laboratory-level  program  reviews  of  ongoing  research. 

EPA  also  established  a  mandatory  quality  assurance  program 
in  June  1979  for  all  offices  engaged  in  monitoring  and  measurement 
efforts.  The  program's  two  major  objectives  are  to  provide 
decisionmakers  with 

— a  clear  understanding  of  the  quality  of  data  for  decisions 
and 

--guidance  and  criteria  for  defining  the  quality  of  data 
for  program  implementation. 


Exploratory  research 


The  Congress  responded  to  the  criticism  that  EPA  lacked  an 
effective  long-term  exploratory  research  program  by  enacting 
Public  Law  95-155,  the  Environmental  Research,  Development  ana 
Demonstration  Authorization  Act  of  1978.  Section  b(a)  of  the 
act  required  EPA  to  establish  a  continuing  long-term  research 
program.  The  law  also  required  that  a  minimum  of  15  percent  of 
appropriated  environmental  research  and  development  funds  be 
allocated  for  long-term  research.  The  Congress  has  included 


4 


similar  provisions  in  EPA's  research  and  development  authori¬ 
zation  acts  far  fiscal  years  1979,  1980,  and  1981. 

In  response  to  past  criticisms  and  congressional  mandates, 

EPA  established  an  Office  of  Exploratory  Research  and  initiated 
an  institutional  support  program  through  exploratory  research 
centers.  The  Office  of  Exploratory  Research  is  responsible  for, 
among  other  things, 

— establishing  an  organization  focal  point  for  long- 
range  research,  identifying  emerging  problems,  and 
developing  programs  in  response  to  such  problems; 

— developing  the  capability  for  assessing  emerging 
problems,  their  importance  or  impacts,  and  trans¬ 
lating  these  needs  into  resource  allocation 
decisions;  and 

— bringing  together  related  basic  research  programs 

to  maximize  internal  assessment  and  planning  functions. 

The  exploratory  research  center  program  is  designed  to  use 
institutions  with  well-established  expertise  in  a  specific  area; 
these  institutions  will  focus  on  long-term  (3  to  5  years  or 
longer)  exploratory  research  to  provide  the  link  between  basic 
and  applied  research,  as  it  relates  to  EPA's  missions.  Seven 
centers  have  been  established — three  in  fiscal  year  1979  and  four 
in  fiscal  year  1980.  Because  EPA  is  in  the  process  of  developing 
and  modifying  the  various  components  of  the  exploratory  research 
program,  we  believe  it  xs  too  early  for  us  to  evaluate  its 
activities . 

OBJECTIVE,  SCOPE,  AND  METHODOLOGY 

The  objective  of  this  review  was  to  determine  the  extent 
to  which  EPA  is  planning  and  performing  research  that  is  valid 
and  useful  in  fulfilling  the  Agency's  mission.  We  wanted  to 
evaluate  the  degree  to  which  EPA  took  action  on  its  June  30,  1978, 
report  to  the  President  and  the  Congress  in  which  it  identified 
major  problems  and  solutions  for  EPA's  research  planning  and 
management. 

Unless  otherwise  specified,  this  report  deals  with  the 
approximately  85  percent  of  the  ORD  budget  that  does  not  consti¬ 
tute  exploratory  research. 

We  conducted  our  review  at  EPA  headquarters  and  the  following 
laboratories : 

— Environmental  Sciences  Research  Laboratory,  Research 
Triangle  Park,  North  Carolina; 


— Health  Effects  Research  Laboratory,  Research  Triangle 
Park,  North  Carolina; 

— Health  Effects  Research  Laboratory;  Cincinnati,  Ohio;  and 

— Municipal  Environmental  Research  Laboratory,  Cincinnati, 
Ohio. 

These  laboratories  were  selected  because  they  represent  the 
majority  of  research  conducted  for  5  of  the  9  EPA  program  offices 
and  7  of  the  14  research  committees.  Furthermore,  the  work  con¬ 
ducted  and  managed  at  the  selected  laboratories  comprises  approx¬ 
imately  29  percent  of  the  total  EPA  research  budget  for  all  Is 
laboratories . 


1981  ORD  Expenditures 

Extramural  Inhouse 


Total 


Environmental  Sciences 

Research  Laboratory,  N.C. 

§  19,817.0 

— ( thousands )- 

$  5,984.5 

$  25,801.5 

Health  Effects  Research 
Laboratory,  N.C. 

13,778.4 

14,338.6 

28,117.0 

Health  Effects  Research 
Laboratory,  Ohio 

11,815.5 

b , 259 . 4 

18,074.9 

Municipal  Environmental 

Research  Laboratory,  Ohio 

22,155.8 

8,055.0 

30,210.8 

Total  for  4 
laboratories 

All  EPA 

laboratories 

$  b7,56b.7 

$  34 , b37 . 5 

$102,204.2 

$250,835.7 

$103,592.9 

$354, 428. b 

Percent  of  total 

27% 

33% 

29% 

We  interviewed  persons  in  the  following  five  program  offices 
to  obtain  their  views  on  research  planning  and  management  in  EPA; 

— Office  of  Air  Quality  Planning  and  Standards. 

— Office  of  Drinking  Water. 

— Office  of  Mobile  Source  Air  Pollution  Control. 

— Office  of  Water  Program  Operations. 

— Office  of  Water  Regulations  and  Standards. 


b 


We  also  discussed  the  research  program  with  officials  from 
the  following  seven  research  committees: 

— Drinking  water. 

— Gases  and  particles. 

— Hazardous  air  pollutants. 

— Mobile  sources. 

— Municipal  wastewater  and  spill  prevention. 

— Oxidants. 

— Water  quality. 

We  interviewed  project  officers,  EPA  and  non-EPA  researchers, 
and  users  of  randomly  selected  research  projects  to  better  under¬ 
stand  the  communication  process  between  ORD  and  its  clients.  We 
also  examined  documents,  procedures,  and  practices  pertaining  to 
research  committee  operations  and  the  peer  review  and  quality 
assurance  programs. 

We  interviewed  officials  in  the  following  agencies  to  obtain 
their  feelings  about  research  quality  and  usefulness:  the  Food 
and  Drug  Administration,  the  National  Academy  of  Sciences,  the 
National  Institute  for  Occupational  Safety  and  Health,  and  the 
National  Institute  of  Environmental  Health  Sciences.  We  also 
discussed  procedures  for  assuring  research  quality  with  aca¬ 
demicians  from  the  University  of  Maryland  and  the  University  of 
Cincinnati  and  company  officials  from  PEDCO  Environmental,  Inc., 
and  ARMCO  Steel  and  Kettering  Laboratory  in  Cincinnati.  We 
also  discussed  EPA  research  quality  with  the  Automobile 
Manufacturers  Association  and  the  Environmental  Defense  Fund. 

Finally,  we  discussed  the  matters  contained  in  this  report 
with  ORD's  Deputy  Director,  Office  of  Research  Management  and 
other  ORD  officials.  Where  appropriate,  their  comments  have  been 
incorporated  in  the  final  report. 
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CHAPTER  2 


RESEARCH  PLANNING  HAS  IMPROVED 
BUT  PROBLEMS  REMAIN 

Communications  between  research  performers  and  users  have 
improved  and  responsiveness  of  research  findings  to  program  needs 
is  better  than  in  the  past  due  primarily  to  the  establishment  of 
research  committees.  However,  despite  these  improvements,  the 
research  committees  have  not  always  been  able  to  achieve  agreement 
between  ORD  and  the  various  program  offices  concerning  research 
strategies  and  specific  research  projects  to  be  performed. 

Research  strategies  are  intended  to  be  a  commitment  between 
ORD  and  program  offices  that  identifies  and  ranks  research  to 
be  conducted  over  a  period  of  several  years.  Although  charged 
with  developing  such  strategies,  only  three  of  the  seven  commit¬ 
tees  we  reviewed  had  produced  finalized  and  approved  strategy 
documents.  None  were  approved  in  time  for  the  fiscal  year  1982 
budget  cycle. 

Disagreement  about  some  currently  planned  research  also 
continues.  The  relevance  to  program  office  needs  of  many 
currently  planned  research  projects  has  been  questioned  by 
several  research  committees  without  satisfactory  resolution. 
Committees  have  no  authority  to  require  solution  of  these  prob¬ 
lems,  and  ORD  has  little  incentive  to  otherwise  resolve  them. 
Unless  ORD  and  the  program  offices  formally  agree  through  the 
research  committees  that  planned  research  will  reasonably  meet 
the  Agency's  priority  regulatory  needs,  inferior  and  untimely 
research  results  may  continue  as  a  criticism  of  ORD  research. 

SLOW  PROGRESS  IN  DEVELOPING 
RESEARCH  STRATEGIES 

EPA  has  made  significant  strides  in  overcoming  a  lack  of 
research  strategy  by  instituting  research  committees  and  requiring 
that  they  annually  develop  research  strategy  documents.  However, 
slowness  in  developing  these  strategy  documents  has  hampered  their 
effectiveness.  Of  the  seven  research  committees  we  reviewed,  none 
had  produced  strategy  documents  in  time  for  the  1982  fiscal  year 
budget  cycle.  Although  some  strategy  documents  have  since  been 
completed  and  others  are  in  draft  form,  they  vary  greatly  in 
scope,  detail,  and  presentation  of  program  offices  research  needs 
and  ORD-planned  activity. 

Few  completed  strategies 

Although  some  committees  have  successfully  outlined  program 
office  needs  and  ORD's  planned  response  over  a  short-term  period, 
few  have  actually  developed  a  comprehensive,  long-term  research 
strategy.  Several  ORD  managers  said  that  strategies  are  integral 
to  any  further  discussions  between  researchers  and  program  office 
users  about  the  research  that  should  be  performed  to  fulfill 


those  strategy  needs.  They  said  that,  without  completed  strate¬ 
gies,  further  discussions  about  the  specific  research  to  be  per¬ 
formed  were  severly  impeded,  since  there  was  no  assurance  that 
both  groups  had  reached  agreement  on  a  common  goal . 

In  March  1979  research  committees  were  asked  to  have  updated 
strategy  documents  available  in  time  for  the  fiscal  year  1982 
budget  preparation  in  the  spring  of  1980.  Of  the  seven  strategy 
documents  we  reviewed,  only  three  received  joint  program  office/ 
ORD  approval  during  1980-81.  None  of  those  were  completed  in 
time  for  use  in  the  budget  process.  As  of  April  1981,  the  remain¬ 
ing  four  committees  still  had  not  completed  their  strategies — 
three  were  in  draft  form  while  one  was  not  developed  in  any 
usable  form. 

According  to  research  committee  officials,  research  planning 
disagreements  between  ORD  and  the  program  offices  were  the  pri¬ 
mary  reasons  strategies  were  not  completed.  Resolving  these 
disagreements  was  hampered,  according  to  these  officials, 
because  key  personnel  retired  or  resigned  during  the  recent 
change  of  administration. 

The  Stanford  Research  Institute  International  (SRI)  also 
identified  problems  in  developing  research  strategies.  In  its 
December  1980  report  1/  SRI  noted  that  the  development  of  multi¬ 
year  research  strategies  by  research  committees  has  been  slow  and 
those  that  have  been  developed  do  not  effectively  extend  beyond  2 
years  because  of  such  factors  as  the  continuing  need  to  prepare 
the  next  fiscal  year's  budget  and  the  program  offices'  emphasis 
on  immediate  needs. 

Varying  scope  and  detail 
of  strategy  documents 

While  recognizing  that  committees  should  have  appropriate 
flexibility  in  developing  their  strategies,  the  March  1979 
General  Guidance  for  research  committees  outlined  the  minimum 
requirements  that  should  be  satisfied  in  every  document, 
including : 

— Legislative  mandate  and  authority. 

— Regulatory,  regional,  and  enforcement  needs  or  require¬ 
ments  . 

— ORD  response  to  research  requirements. 

— Timetable  and  expected  products. 

— Resources  for  each  year  covered  in  the  document. 


l/"Health  Effects  Research  and  Standard  Setting  at  EPA, "  SRI, 
Dec.  1980. 
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We  reviewed  seven  research  committee  strategy  documents 
(for  those  which  had  not  been  finalized  we  reviewed  draft 
documents)  and  found  that  they  varied  greatly  in  scope,  detail, 
and  presentation  of  program  office  needs  and  ORD-planned 
activity.  The  documents  varied  in  length  from  21  pages 
(hazardous  air  pollutants  draft)  to  143  pages  (drinking  water). 

Program  offices'  statements  of  need  and  ORb  responses 
differ  in  scope  and  presentation  from  strategy  to  strategy. 

The  drinking  water  strategy,  for  example,  contains  detailed 
program  and  regional  office  needs  by  contaminant  and  research 
area;  it  includes  detailed  laboratory  by  laboratory  responses 
that  specify  individual  projects  and  scheduled  completion 
dates.  At  the  other  extreme,  the  hazardous  air  pollutants 
strategy  document  does  not  discuss  specific  program  office  needs 
but  instead  outlines  general  areas  and  uses  of  research  to  be 
performed. 

The  degree  of  long-term  strategic  planning  varies  greatly 
among  these  strategy  documents.  The  water  quality  document,  for 
example,  contains  a  strategic  outlook  in  the  form  of  a  10-page 
section  entitled  "Projected  Agency  Policies  and  Priorities  in 
the  1980 ‘s."  Most  other  documents  mainly  dealt  with  research 
planning  over  a  1-3  year  period  with  minimal  input  given  to 
long-term  planning.  The  draft  oxidants  strategy  document,  for 
example,  represents  an  overview  of  the  current  ongoing  research 
program  with,  in  our  opinion,  no  real  strategic  planning  included. 

In  its  January  1980  report  entitled  "Evaluation  of  Pilot 
Research  Committees,"  EPA's  Program  Evaluation  Division  noted  that 
research  committees  "must  *  *  *  begin  to  perform  truly  strategic 
research  planning."  The  report  notes  that  both  short-term  detailed 
planning  and  longer  term  strategic  planning  are  important  compo¬ 
nents  of  strategy  documents.  It  states  that  the  level  of  detail 
is  important  in  that  it  fosters  an  understanding  within  ORD  as 
to  the  explicit  nature  of  the  research  needs  of  the  program 
offices  and  allows  them  a  greater  degree  of  certainty  as  to 
the  research  being  conducted  by  ORD.  The  report  also  recommends 
that  the  committees  include  strategic  planning  as  a  necessary 
major  component  of  their  research  planning  function. 

The  Science  Advisory  Board  has  also  criticized  EPA's  stra¬ 
tegic  planning.  The  Board  provides  independent  scientific  and 
technical  advice  on  numerous  topics  within  the  purview  of  EPA's 
regulation  and  research.  In  its  assessment  of  the  pesticides 
research  strategy,  the  Board  was  critical  of  the  effort  put 
into  the  strategy.  While  noting  that  the  use  of  the  research 
committee  structure  resulted  in  greater  coordination  and  coopera¬ 
tion  between  the  Office  of  Pesticide  Programs  and  ORD,  the  Board 
found  that  the  strategy  was  poorly  written  and  incomplete  in 
that  it  did  not  sufficiently  address  several  important  areas 
of  research.  The  Board  also  commented  that  errors  of  fact  were 
numerous  in  the  document  and  it  lacked  clarity  and  organization. 
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EPA  has  taken  action  that  may  improve  the  development  of 
multiyear  strategic  planning  in  the  future.  In  its  April  1981 
guidance  for  research  committees,  EPA  requires  that  all  commit¬ 
tees  prepare  a  5-10  page  multiyear  strategy  that  will  be  used 
as  a  basis  for  formulating  the  fiscal  year  1983  budget  proposals. 
According  to  the  acting  director,  program  coordination  staff, 

ORD,  the  commi' tees  can  include  more  information  in  their  docu¬ 
ments  if  they  desire;  the  5-10  page  requirements  will  only  serve 
to  standardize  the  basic  strategy  approach  of  all  documents. 

PROBLEMS  IN  PLANNING 
RESPONSIVE  RESEARCH 


Research  committees  are  the  only  formalized  structure  for 
obtaining  and  communicating  research  user's  needs  to  those  who 
perform  research.  The  research  committee  system  is  designed  to 
assure  that  planned  and  ongoing  research  projects  are  consistent 
with  approved  strategy  documents,  are  relevant  to  EPA's  priority 
research  needs,  and  will  provide  timely  outputs  for  effective 
regulatory  decisionmaking. 

Our  review  of  EPA's  research  planning  process  indicated  that 
program  offices  were  not  always  satisfied  with  planned  or  ongoing 
research.  ORD  treats  research  committee  ideas,  comments,  and 
suggestions  as  advisory  input  only,  with  little  direct  incentive 
and  no  requirement  to  be  responsive  to  these  expressed  needs. 

We  found  the  following  problems  inhibit  committee  efforts  to 
effectively  plan  responsive  research; 

— Communication  between  research  users  and  ORD  through 
the  committee  process  has  not  always  been  effective  in 
assuring  that  useful  research  projects  and  outputs 
will  be  performed. 

— EPA  has  not  implemented  its  plan  to  provide  direct  incen¬ 
tives  to  researchers  to  be  responsive  to  regulatory  needs. 

— EPA  has  not  established  any  mechanism  requiring  agreement 
before  research  is  started  that  the  approach  and  timing 
of  planned  projects  are  reasonable  to  meet  intended 
needs . 

Improved  communications  have 
not  assured  useful  research 


Strategy  documents  and  laboratory  output  plans  are  EPA's 
mechanisms  for  communicating  research  needs  and  approaches. 
Shortcomings  in  their  preparation  and  review  can,  however, 
prevent  assurance  that  useful  research  projects  and  outputs 
will  be  performed. 

Laboratory  output  plans  are  composed  of  two  parts.  Part 
one  contains  ORD  guidance  on  the  major  research  objectives. 
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major  outputs,  and  expected  completion  dates.  Part  two  is  pre¬ 
pared  by  the  laboratories  and  contains  more  specific  information 
about  the  research  area  such  as  approach,  rationale,  completion 
dates,  and  outputs.  Each  research  area  may  be  comprised  of 
as  many  as  15-20  related  research  projects.  To  further  explain 
the  major  research  areas,  some  laboratories  were  recently  asked 
to  prepare  one-page  descriptions  for  each  research  project. 

These  summaries  contain  the  title,  objective,  approach,  rationale, 
resources,  milestones,  and  laboratory  contact  for  each  project. 
This  total  package  is  then  sent  to  the  research  committees  for 
review. 

Our  review  of  the  fiscal  year  1981  research  planning  process 
indicated  that  the  relevance  of  many  currently  planned  and  on¬ 
going  projects  was  questioned  by  several  research  committees 
without  satisfactory  resolution.  Projects  were  questioned  for 

— lack  of  sufficient  detail  in  ORD's  plans  to  relate 
project  objectives  to  program  needs, 

— incongruities  between  project  size  and  priority 
and  the  planned  ORD  expenditures,  and 

— lack  of  defined  interrelationships  between  planned 
and  ongoing  projects . 

Still  other  questions  were  raised  by  research  committees  con¬ 
cerning  vital  regulatory  needs  that  appear  not  to  have  been 
addressed  by  ORD's  research  plans.  These  concerns  were  not 
always  resolved  satisfactorily. 

Of  the  seven  research  committees  we  reviewed,  only  two  (water 
quality  and  drinking  water)  were  satisfied  that  the  committee 
process  had  produced  laboratory  output  plans  that  would  reasonably 
meet  their  research  needs.  The  program  office  cochairman  of 
the  municipal  wastewater  and  spill  prevention  research  commit¬ 
tee  was  comfortable  with  ORD's  response  to  the  committee's  review 
comments.  He  believed  the  research  committee  structure  has 
been  very  helpful  in  generating  communication  for  research  plan¬ 
ning;  however,  he  said  that,  to  get  specific  research  needs 
fulfilled,  it  continues  to  be  a  "case  of  who  you  know."  The 
following  summarizes  the  concerns  expressed  by  the  other  four 
committees . 

Gases  and  particles  research  committee 

This  committee's  program  office  cochairman  said  that,  as 
of  February  26,  1981,  the  laboratory  output  plans  had  not  been 
finalized.  As  a  result,  he  did  not  know  what  projects  were 
actually  funded  for  fiscal  year  1981  even  though  the  fiscal  year 
was  almost  half  over.  Part  of  the  delay  for  fiscal  year  1981 
resulted  from  EPA  not  receiving  its  appropriation  until  December 
1980.  Still,  as  the  program  office  cochairman  noted,  it  is  oif- 
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ficult  for  the  program  office  to  plan  for  fiscal  year  1982  research 
work  when  it  does  not  know  what  was  approved  for  the  prior  year. 

In  reviewing  this  research  area's  draft  laboratory  ohtput 
plans,  the  committee  identified  various  problems.  These  {included 
such  matters  as 

— various  inconsistencies  and  omissions,  which  make  it 
unclear  in  many  instances  exactly  what  a  research 
project  would  produce; 

— lack  of  plans  for  certain  program  needs; 

— limited  usefulness  of  outputs  in  many  cases;  and 

— many  items  that  were  not  responsive  to  regulatory 
needs  and  schedules. 

For  example,  one  project  that  was  given  the  highest  priority 
rating  by  the  research  committee  was  never  funded  by  ORD.  The 
project  would  determine  whether  sulfur  dioxide  is  a  cocarcinogen 
in  animals.  The  program  office  indicated  that  delaying  this 
project  for  another  year  or  two  could  mean  that  it  will  not  be 
completed  in  time  for  the  next  sulfur  dioxide  standard  review. 

This  review  is  congressionally  mandated  and  requires  EPA  to 
periodically  assess  the  scientific  validity  of  its  standards. 

The  committee  cochairman  stated  that  he  was  not  totally 
satisfied  with  ORD's  response  to  his  comments  because  very  little 
change  was  made  to  the  plans. 

Oxidant  research  committee 

As  of  February  17,  1981,  this  research  area's  laboratory 
output  plan  had  not  been  finalized.  Based  on  its  review  of  the 
draft  plans,  the  committee  found 

— some  projects  to  be  of  low  priority  because  they 
could  duplicate  other  work  or  did  not  appear 
useful, 

— certain  project  information  was  unclear,  and 

— several  additional  projects  were  not  included  but 
of  interest  to  the  program  office. 

For  example,  the  program  office  committee  cochairman  recommended 
that  ORD's  project  for  developing  a  photochemical  box  model  to 
measure  ozone  air  quality  be  delayed,  reduced,  or  cancelled.  The 
committee  stated  that  it  has  no  plans  at  this  time  to  use  this 
model  over  the  next  several  years.  The  laboratory,  however,  is 
undertaking  the  project  in  spite  of  the  committee's  opposition. 


The  committee  cochairman  said  that  he  was  not  satisfied  with 
ORD's  responsiveness  to  the  program  office  needs.  He  did  say, 
however,  that  he  believed  ORD  management  was  trying  to  do  what 
it  could,  but  he  felt  little  change  would  occur  to  the  plans 
because  of  budget  restrictions.  He  said  that  research  needs  not 
addressed  in  fiscal  year  1981  plans  would  probably  be  included 
in  future  years’  plans  to  the  extent  possible. 

Mobile  source 
research  committees 


Laboratory  output  plans  for  this  research  area  were  not 
finalized  as  of  March  17,  1981.  According  to  the  program  office 
committee  cochairman,  the  draft  plans  did  not  satisfy  all  the  pro¬ 
gram  office  needs.  Six  projects  totaling  $1,387,000  were  acceptable 
but  21  other  projects  with  a  proposed  cost  of  $7,255,600  were 
not.  In  many  cases,  the  unapproved  projects  were  wanted  but 
the  committee  could  not  place  them  on  the  approved  listing  until 
a  better  description  of  what  ORD  was  planning  was  submitted. 

Some  of  the  committee's  other  comments  were  that 

— some  projects  were  of  low  priority  and  should  not  be 
funded, 

— approaches  for  conducting  research  generally  were  missing 
or  lacked  specificity,  and 

— objectives  of  some  projects  were  not  clearly  stated. 

The  program  office  committee  cochairman  advised  ORD  of  his 
concern  that  ORD  may  spend  fiscal  year  1981  mobile  source  funds 
for  work  not  requested  by  the  program  office.  He  said  that  such 
action  would  result  in  a  needless  waste  of  Government  funds.  For 
example,  the  program  office  needed  a  brief  evaluation  of  the  real 
time  particulate  monitor  developed  by  Ford  and  General  Motors. 

The  program  office  committee  cochairman  said  that,  since  neither 
company  seems  confident  that  these  monitors  will  be  acceptable, 
the  priority  for  this  work  should  be  low  and  the  $110,000  funding 
appears  too  high  considering  this  project's  priority.  He  also 
said  that  this  work  should  be  held  in  abeyance  and  will  be 
approved  later  if  funding  permits  after  the  entire  plan  is 
reviewed . 

The  program  office  committee  cochairman  was  not  satisfied 
with  ORD's  responsiveness  to  committee  comments.  He  said  that 
because  of  ORD's  poor  responsiveness,  the  program  office  had 
decreased  resources  committed  to  the  research  committee  effort. 

Hazardous  air  pollutants 

research  committee 


Laboratory  output  plans  for  this  area  were  finalized  as  of 
January  30,  1981.  Although  agreement  was  reached  on  many  aspects 
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of  the  plan,  the  committee  did  identify  some  problems.  It  found 
that  four  projects  rated  as  lower  priority  continue  to  receive 
funds  although  they  are  only  marginally  useful.  For  example,  a 
project  concerning  bioassay  support  for  identifying  airborne  car¬ 
cinogens  was  rated  as  low  priority  but  still  is  being  conducted. 

The  program  office  committee  cochairman  said  that  this  work  has 
also  been  performed  at  several  sites  and  a  final  report  on  the 
findings  is  due  soon.  He  said  that  after  reviewing  the  final 
report  and  when  better  methods  have  been  developed  for  collecting 
and  analyzing  gaseous  samples,  this  type  of  study  may  merit 
higher  priority. 

The  program  office  committee  cochairman  believes  that  budget 
restrictions  also  partially  affected  ORD's  ability  to  be  more 
responsive. 

Research  incentive  system 
short-lived 

Researchers  and  regulators  often  disagree  about  the  best  way 
to  satisfy  EPA's  research  needs.  Because  of  different  profes¬ 
sional  perspectives  based  on  experiences  and  demands  on  research¬ 
ers  and  regulators,  an  inherent  set  of  tensions  is  created,  which 
results  in  complaints  of  unresponsive  research  results.  To  resolve 
this  inherent  dichotomy  between  researchers  and  regulators,  EPA 
proposed  to  establish  direct  incentives  to  promote  more  responsive 
research  by  ORD.  However,  the  incentive  system  was  never  imple¬ 
mented. 

Differing  research  perspectives 

Various  studies  have  long  recognized  the  inherent  differences 
in  perspectives  between  researchers  and  regulators.  The  December 
1980  SRI  study  noted  that 

"communication  between  these  two  groups  is  made  difficult 
by  their  different  interests  and  emphasis:  regulatory 
personnel  are  results-oriented  and  are  less  sensitive  to 
the  difficulties  and  time  demands  of  research,  whereas 
scientists  are  process-oriented  and  are  less  sensitive 
to  providing  specific  results  to  meet  deadlines." 

In  interviews  with  many  ORD  and  program  office  staff  members,  we 
similarly  found  that  researchers  and  regulators  often  differ 
substantially  in  their  ideas  on  how  best  to  satisfy  EPA's  research 
needs . 

Although  the  National  Academy  of  Sciences  recognized 
similar  differences  in  research  perspectives,  it  concluded  that 
"because  EPA  is  primarily  a  regulatory  agency  with  limited 
resources,  the  principal  reason  for  its  research  must  be  to 
support  decisionmaking." 
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Research  incentive  system 

Recognizing  this  inherent  dichotomy  between  the  roles  of 
researchers  and  regulators,  EPA  established  a  pilot  research 
incentive  system  in  March  1978  whereby  a  portion  of  the  research 
funds  would  be  alloted  to  program  and  regional  offices  rather  than 
ORD.  EPA  believed  this  would  establish  a  client/contr ,ctor  rela¬ 
tionship  between  researchers  and  regulators  that  would  require 
them  to  work  together  to  avoid  unresponsive  research  results. 

Under  the  system,  program  offices  would  identify  research 
areas  they  believed  they  should  more  closely  manage  because 
those  areas  are  key  to  the  success  of  their  regulatory  efforts 
in  the  nearterm.  These  offices  would  then  negotiate  with  ORD 
to  conduct  the  needed  research  within  its  laboratories  or  to 
provide  contract  management  services  for  the  projects.  Al¬ 
ternatively,  the  program  offices,  after  consulting  with  ORD, 
could  contract  directly  with  outside  firms  to  conduct  the  desired 
research.  By  allowing  ORD  to  choose  whether  or  not  to  "bid"  on 
key  projects,  the  system  would  allow  ORD  to  set  its  prices  com¬ 
petitively  in  order  to  develop  inhouse  research  capability. 

According  to  EPA,  the  system  would  provide  a  mechanism  to 
encourage  ORD  to  be  responsive  to  regulatory  research  needs 
and  encourage  program  and  regional  offices  to  take  seriously 
their  responsibilities  for  clearly  articulating  their  research 
needs  and  for  following  the  progress  of  key  research  projects. 

The  intent  of  establishing  a  research  incentive  system  was  to 
enhance  the  impact  of  the  research  committee  system  in  promoting 
openness  and  responsiveness  in  communications  between  research 
and  program  managers . 

The  system,  however,  was  never  funded,  according  to  the 
chief  of  EPA's  program  evaluation  branch  in  the  office  of 
Planning  and  Evaluation.  EPA  decided  not  to  implement  the 
system  because  it  believed  that  the  research  committee  system 
is  effectively  promoting  mutual  understanding  and  responsiveness. 

The  system  for  resolving 
conflicts  is  inadequate 

As  originall .  established,  each  research  committee  had  the 
responsibility  for  serving  as  the  primary  forum  for  resolving 
research  issues  encountered  by  the  program  offices,  the  regional 
offices,  or  ORD  during  the  planning  and  implementation  years. 
Realizing  that  total  agreement  on  research  to  be  performed  may 
not  be  achieved  within  the  research  committee  structure,  in  1978 
EPA  proposed  establishing  the  research  oversight  committee  to 
resolve  such  disagreements.  This  committee,  which  was  to  be 
comprised  of  the  EPA  Assistant  Administrators,  never  became 
operational.  Instead,  ORD  established  a  research  council  com¬ 
prised  solely  of  ORD  management  to  resolve  planning  disagreements. 


It 


According  to  the  acting  director  of  ORD's  program  coordina¬ 
tion  staff,  the  research  council  addresses  general  issues  that 
affect  ORD’s  overall  operations.  He  told  us  that  a  program  office 
cochairman  whose  specific  research  needs  were  not  satisfied 
through  the  research  committee  system  can  appeal  to  his  or  her 
Assistant  Administrator.  The  program  office  Assistant  Adminis¬ 
trator  will  then  outline  his  or  her  concerns  to  the  Assistant 
Administrator  for  Research  and  Development,  who  will  make  the 
final  determination  on  the  matter.  In  essence,  research  committee 
concerns  about  what  research  ORD  should  undertake  are  appealed  to 
ORD  management. 

One  research  committee — mobile  source — attempted  to  develop 
its  own  system  for  resolving  research  planning  disagreements, 
but,  according  to  its  cochairman, ORD  never  adhered  to  the  system. 
The  system  stipulated  that  whenever  disagreements  arose,  the 
program  office  decisions  would  rule  or  control  up  to  80  percent 
of  the  funding  for  mobile  source  research  and  ORD  laboratories 
would  control  the  remaining  20  percent.  This  cochairman  said 
that  he  would  be  satisfied  with  a  50-50  rule  if  ORD  would  honor 
it. 

In  its  December  1980  report,  SRI  reported  that  although  an 
appeal  process  has  been  articulated  “it  has  never  been  used  in 
the  air  research  programs;  the  process  has  always  reached  a 
satisfactory  adjudication  at  a  lower  level."  The  mobile  sources 
committee  cochairman  questioned  whether  this  was  entirely 
accurate.  He  said  that  underlying  the  adjudication  process  are 
several  factors.  In  addition  to  a  hopeful  wait-and-see  attitude, 
program  office  committee  cochairmen  do  not  want  to  appear  too 
negative  about  ORD's  efforts  to  plan  and  produce  responsive 
research,  which  has  improved  substantially.  Also,  because 
research  committee  comments  are  only  advisory  in  nature  and  ORD 
has  final  authority  over  the  research  planned  to  be  performed, 
appeals  are  not  viewed  by  some  cochairmen  as  effective  ways 
of  enhancing  ORD/program  office  communication  and  cooperation. 

Program  office  members  of  research  committees  we  contacted 
agreed  that  ORD's  responsiveness  continues  to  depend  heavily  on 
informal  relationships  with  ORD  staff.  Although  they  believe 
the  research  committee  system  has  caused  substantial  improve¬ 
ments,  they  also  told  us  that  responsive  research  planning  still 
has  not  been  achieved  in  some  instances  and  is  highly  ques¬ 
tionable  in  others.  The  December  1980  SRI  report  stated  that 

"the  mechanisms  for  this  responsiveness  appear  to 
be  informal  rather  than  a  formal  consideration  and 
decision  by  the  relevant  research  committee." 


Need  to  establish  mechanism  for 
requiring  agreement  on  research 


>lans 


In  its  January  1980  report,  EPA's  Program  Evaluation  Divi¬ 
sion  recommended  that  committees  reach  agreement  on  the  content. 
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conduct,  and  timing  of  critical  program-related  research  to  the 
point  that  a  client/contractor  relationship  between  ORD  ana 
the  program  office  is  established.  We  made  a  similar  recommen¬ 
dation  in  our  October  1980  report  entitled  "Promising  changes 
Improve  EPA's  Extramural  Research;  More  Changes  Needed"  (CED-81-6), 
which  stated  that  the  Administrator  should: 

" — Establish  procedures  that  require  regulatory  offices 
and  laboratories  to  agree  before  work  is  started  that 
the  approach  and  timing  of  research  projects  are 
reasonable  to  meet  intended  needs." 

EPA  disagreed  with  the  recommendation.  It  stated  that  the 
jointly  chaired  research  committees  now  perform  project  planning; 
their  review  of  both  the  zero-based  budget  request  and  the 
laboratory  output  plans  assure  program  office  concurrence.  It 
said  that  the  regulatory  offices  can  and  do  notify  research 
committees  when  there  is  a  shift  in  their  needs,  and  adjustments 
are  made  by  ORD.  It  said  that  project  development,  on  the  other 
hand,  is  the  internal  management  prerogative  of  ORD's  line  managers. 
A  requirement  for  last-minute  approval  of  the  approximately 
1,800  new  projects  in  a  given  year  just  before  they  are  about 
to  begin — presumably  after  all  lengthy,  administrative  work  has 
been  done — is  tantamount  to  agreeing  to  what  has  previously  been 
agreed  upon;  it  is  infeasible  and  managerially  undesirable. 

This  would  inevitably  result  in  the  slowdown  of  research  programs, 
as  well  as  a  vast  increase  in  bureaucratic  papershuffling. 

We  disagree  with  EPA's  response  to  our  recommendation.  As 
discussed  earlier,  agreement  is  not  always  reached  concerning  the 
laboratory  output  plans  and  not  all  research  committee  cochairmen 
are  satisfied  with  ORD's  degree  of  responsiveness  to  their 
expressed  needs.  Unless  agreement  is  reached  on  the  conduct, 
content,  and  timing  of  planned  research,  we  believe  untimely  and 
less  than  useful  research  results  will  continue. 

In  discussing  this  recommendation  with  selected  ORD  laboratory 
directors  and  research  committee  cochairmen  and  representatives, 
we  were  told  that  it  would  be  reasonable  to  implement  such  a  proce¬ 
dure  for  a  portion  of  ORD's  research  budget  but  that  agreement 
would  probably  not  be  reached  on  ORD's  total  planned  expenditures 
in  any  given  year. 

Most  program  office  cochairmen  of  research  committees  we 
contacted  did  not  want  total  control  of  ORD's  research  budget; 
however,  most  did  believe  that,  as  a  minimum,  50  percent  of 
ORD's  planned  expenditures  should  be  expressly  ratified  by  the 
research  committees  before  the  research  is  started.  Others 
believed  60  percent  or  more  should  be  handled  in  this  way. 

For  example,  a  portion  of  EPA's  research  budget  would  be 
allocated  to  the  research  committee.  This  portion — say  50  to 
60  percent  of  ORD's  total  research  budget — would  require  express 
research  committee  agreement  on  the  conduct,  content,  and  timing 
of  research  projects  before  the  work  is  started,  of  the 


remaining  ORD  budget,  15  percent  or  more  would  be  used  foi  the 
congressionally  mandated  anticipatory  research.  ORD  wouJa 
have  final  authority  over  the  remaining  research  budget 
after  considering  the  proyram  offices'  and  research  committees' 
comments  and  suggestions. 

In  its  deliberations  before  passing  Public  Law  95-155, 
the  Senate  Committee  on  Environment  and  Public  Works  considered 
requiring  that  60  percent  of  authorized  research  funas  be  ear- 
marxed  for  activities  funded  through  the  program  offices  rather 
than  ORD.  This  action  was  not  proposed  by  the  committee  nor 
subsequently  acted  on  by  Congress  on  the  basis  that  EPa' s  top 
management  had  recently  changed  and  was  studying  the  problem 
of  coordination  with  an  intent  to  resolve  it. 

CONCLUSIONS 

EPA  has  made  significant  strides  in  overcoming  a  lack  of 
research  strategy  by  instituting  research  committees  and  requiring 
them  to  annually  develop  strategy  documents.  However,  slowness 
in  developing  these  documents  has  hampered  their  effectiveness. 

Without  strategies,  further  discussions  between  ORD  and 
the  program  offices  about  the  specific  research  to  be  performed 
are  severely  impeded.  Additionally,  though  some  committees 
have  been  somewhat  successful  in  outlining  program  office 
needs  and  ORD's  planned  response  over  a  short-term  period,  few 
have  actually  developed  a  comprehensive  long-term  research 
strategy . 

To  assure  that  research  strategies'  provide  a  programmatic 
context  about  which  major  goals  and  objectives  can  be  identi¬ 
fied,  coherent  plans  developed  and  implemented,  and  results 
evaluated,  all  research  committees  need  to  annually  develop 
finalized  strategy  documents.  These  documents  should  include 
both  long-term  strategic  plans  and  the  shorter  term,  more 
detailed  look  at  program  office  needs  and  planned  ORD  responses. 

Communications  between  ORD  and  proyram  offices  have 
improved,  and  responsiveness  of  research  findings  to  proyram 
needs  is  better  than  in  the  past  due  primarily  to  the  establish¬ 
ment  of  research  committees.  However,  despite  these  improve¬ 
ments,  the  research  committees  in  many  cases  have  not  been  able 
to  achieve  agreement  between  ORD  and  the  various  proyram  offices 
concerning  research  strategies  and  research  to  be  performed. 

Enhanced  communication  has  not  proven  to  be  an  effective 
measure  for  assuring  useful  research,  although  it  is  an  integral 
part  of  the  process.  Researchers  lack  inherent  incentives  to 
be  responsive  to  regulatory  needs,  and  EPA  has  not  fully  imple¬ 
mented  its  plan  to  provide  such  incentives  directly.  Appealing 
decisions  outside  the  committee  structure  without  express 
authority  is  viewed  by  many  as  jeopardizing  the  improvements  in 
communication  and  responsiveness  that  have  been  made. 


If  EPA  is  to  maximize  the  use  of  its  limited  research 
budget,  effectively  provide  scientific  data  on  which  to  base  its 
regulatory  decisions,  and  assure  useful  research  results,  agree¬ 
ment  on  the  conduct,  content,  and  timing  of  research  plans  must 
be  achieved  on  program-related  research  before  work  is  started. 
This  could  be  accomplished  by  various  methods.  The  most  pro¬ 
minently  mentioned  methods  are  (1)  the  research  incentive  system 
and  (2)  funding  a  substantial  portion  of  research  projects 
through  research  committees  or  program  offices.  Adopting  either 
alternative  could  accomplish  this  objective. 

RECOMMENDATIONS 

We  recommend  that  the  Administrator  of  EPA 

— require  all  research  committees  to  develop  approved 
strategy  documents; 

— require  strategy  documents  to  include  both  a  long-term 
strategic  outlook  and  a  shorter  term,  more  detailed,  over¬ 
view  of  program  office  needs  and  planned  ORD  responses? 
and 

— establish  procedures  that  require  regulatory  offices  and 
laboratories  to  agree  before  projects  are  started  that  the 
approach  and  timing  of  research  projects  are  reasonable 
to  meet  intended  needs. 

ORD  COMMENTS  AND  OUR  EVALUATION 


ORD  officials  agreed  with  most  of  our  conclusions  and 
recommendations.  They  disagreed  with  our  recommendation  that  EPA 
establish  procedures  that  require' regulatory  offices  and  labora¬ 
tories  to  agree  before  a  project  is  started  that  the  approach  and 
timing  of  research  projects  are  reasonable  to  meet  intended  needs. 
As  discussed  earlier  (see  p.  18),  these  officials  continue  to  be¬ 
lieve  that  general  agreement  on  project  outputs  already  exists  and 
that  a  requirement  for  last-minute  approval  of  the  approximately 
1,800  new  projects  each  year  is  tantamount  to  agreeing  to  what 
has  previously  been  agreed  upon.  According  to  these  officials, 
this  would  inevitably  result  in  a  slowdown  of  research  programs, 
as  well  as  a  vast  increase  in  bureaucratic  papershuffling. 

We  disagree.  As  discussed  earlier,  we  found  that  agreement 
is  not  always  reached  concerning  research  project  outputs  and  not 
all  program  offices  are  satisfied  with  the  laboratory  output  plans 
and  not  all  research  committee  cochairmen  are  satisfied  with  ORD's 
degree  of  responsiveness  to  their  expressed  needs.  We  continue  to 
believe  that  unless  agreement  is  reached  on  the  conduct,  content, 
and  timing  of  planned  research,  untimely  and  less  than  useful 
research  results  will  continue. 
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CHAPTER  3 


MONITORING  RESEARCH  PROCRESS 
AND  PEER  REVIEW  OF  RESEARCH 
PROPOSALS  NEED  IMPROVEMENT 


According  to  a  recent  evaluation  of  research  committees  by 
EPA's  Program  Evaluation  Division,  monitoring  the  progress  of 
research  projects  against  approved  plans  and  strategies  is  an 
integral  step  in  assuring  useful  research  results.  The  evalua¬ 
tion  further  states  that  research  committees  need  to  keep  apprised 
of  significant  modifications  in  the  conduct  of  planned  research. 

We  found,  however,  that  none  of  the  research  committees  we 
reviewed  are  currently  monitoring  research  programs.  Although 
three  committees  intend  to  monitor  projects  in  the  future  using 
existing  information  systems,  we  found  these  systems  inadequate 
mechanisms  for  effectively  monitoring  research. 

EPA  has  been  repeatedly  criticized  for" its  lacx  of  quality 
research  results.  Responding  to  these  criticisms,  EPA  made 
significant  progress  in  assuring  quality  research  by  implementing 
an  external  peer  review  system  and  requiring  research  results 
to  be  published  in  professional  journals.  Despite  these  improve¬ 
ments,  more  needs  to  be  done. 

Additionally,  EPA  has  established  a  mandatory  quality 
assurance  program.  Although  this  program  seems  adequate,  it  is 
too  early  to  gauge  its  effectiveness  because  it  is  not  yet  fully 
implemented . 

INADEQUATE  MONITORING  OF 
RESEARCH  PROGRESS 


Completing  research  as  planned  and  delivering  quality 
results  on  schedule  are  important  measures  of  EPA's  ability 
to  achieve  responsive  research.  Although  research  committees 
are  responsible  for  monitoring  research  progress  against  approved 
strategies  and  plans,  none  of  the  committees  we  reviewed  are 
performing  this  function.  Some  committees  intend  to  monitor 
research  projects  in  the  future  using  existing  ORD  information 
systems.  However,  we  found  these  systems  are  inadequate  to 
effectively  monitor  research  progress  because  they  are  either 
inoperative  or  do  not  contain  sufficient  data. 

Research  committees  are  not 
monitoring  research  progress 

Of  the  seven  xesearch  committees  we  reviewed,  none  have 
been  formally  monitoring  research  progress.  Three  committees 
intend  to  develop  monitoring  systems  in  the  future,  but  the  four 
others  do  not  intend  to  formally  track  projects.  Rather,  these 
four  committees  plan  to  rely  on  the  existing  informal  mechanisms 
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to  track  projects.  They  plan  to  rely  heavily  on  informal  program 
office  input  to  determine  how  well  research  is  progressing  toward 
meeting  their  needs. 


Generally,  research  committee  officials  told  us  that  they  had 
not  yet  had  the  time  to  develop  the  appropriate  mechanisms.  They 
said  that  the  majority  of  their  efforts  to  date  have  been  spent 
developing  detailed  research  strategies,  becoming  aware  of  ongoing 
ORD  research,  and  commenting  on  ORD 1 s  laboratory  output  plans. 

Two  studies  have  recognized  the  importance  of  monitoring 
research  progress.  SRI,  in  its  December  1980  report,  identified 
the  lack  of  an  effective  monitoring  system  as  a  major  problem. 
EPA's  Planning  and  Evaluation  Division,  in  its  January  1980 
report,  stated  that  ".the  committees  were  not  actively  involved 
during  their  first  year  in  overseeing  the  conduct  of  ORD  research 
activities."  This  report  further  noted  that  one  of  the  most 
important  responsibilities  of  the  committees  in  revitalizing  the 
research  planning  system  is  their  function  in  overseeing  how 
planned  research  is  implemented.  We  agree. 

ORD  information  systems  are 
inadequate  for  monltoring~esearch 

Several  ORD  information  systems  are  available  to  help 
research  committees  monitor  research  progress.  These  include  the 
technical  information  plan,  ORD  progress  reports,  and  the  ORD 
Project  Tracking  System  (PTS).  We  found,  however,  that  these 
systems,  as  currently  being  used,  do  not  provide  adequate  data 
to  effectively  monitor  research. 

Technical  information  plan 

The  technical  information  plan  is  intended  to  be  a  compre¬ 
hensive  listing  of  all  outputs  agreed  to  via  the  annual  planning 
process.  According  to  the  senior  management  analyst  in  ORD's 
Technical  Information  Office,  the  system  is  designed  to  confirm 
the  laboratories'  interpretation  of  the  final  laboratory  output 
plans.  Specifically,  it  is  intended  to  list  the  requestor,  the 
type  of  research  output,  the  laboratory  involved,  the  project 
title,  the  project  officer,  and  the  report  completion  date. 

The  system  cannot  be  used  as  a  tracking  system  for  all 
ongoing  research  because  it  includes  only  those  projects 
expected  to  be  completed  and  reported  during  the  current  fiscal 
year.  No  information  is  included  concerning  ongoing  research 
with  expected  outputs  in  future  fiscal  years.  For  example,  the 
technical  information  manager  at  one  laboratory  advised  us  that 
only  20  percent  of  his  laboratory's  ongoing  projects  are 
included  in  that  laboratory's  technical  information  plan. 

Weaknesses  in  data  content  prevent  the  system  from  becoming 
an  effective  tracking  tool.  The  system  does  not  always  list  the 
specific  research  project's  users  and  in  some  cases  lists  no 
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user  at  all.  In  addition,  there  needs  to  be  some  mechanism  for 
informing  both  the  research  committees  and  the  specific  users 
of  the  research  results  of  any  significant  deviations  in  the  con¬ 
duct  of  planned  research.  Of  the  1,074  projects  listed  in  the 
technical  information  plan  for  the  four  laboratories  we  reviewed 
550,  or  51  percent,  had  no  individual  user  identified;  292,  or  27 
percent,  had  only  a  research  committee  or  program  office  identi¬ 
fied  as  the  user;  and  only  208,  or  19  percent,  had  a  user  name 
listed.  Unless  identified  in  the  system,  the  individual  user  may 
not  be  aware  that  a  research  project  is  underway  to  fulfill  his 
or  her  needs. 

Recognizing  these  weaknesses,  one  program  office — the  Office 
of  Air  Quality  Planning  and  Standards — is  developing  a  second 
phase  to  the  technical  information  plan.  It  involves  designating 
a  program  office  staff  member  to  track  research  progress.  The 
objective  is  to  provide  project  tracking  and  ensure  that  reports 
are  submitted  on  time  and  disseminated  to  the  right  program  office 
individuals.  The  system  also  provides  for  feedback  to  the  research 
committees  on  how  well  projects  responded  to  regulatory  needs. 

Progress  reports 

Periodic  progress  reports  are  part  of  ORD's  internal  manage¬ 
ment  reporting  system.  The  content  of  the  reports  includes  the 
project  title  and  number,  principal  investigator,  regulatory 
authority  and  program  support  areas,  project  objectives,  mile¬ 
stones,  and  significant  results.  ORD  has  two  types  of  progress 
reports — quarterly  and  monthly.  The  former  is  issued  by  labora¬ 
tories  every  3  months  and  includes  an  update  of  all  ongoing  pro¬ 
jects.  The  latter  is  issued  monthly  but  contains  highlights  of 
only  selected  ongoing  projects.  Neither  quarterly  nor  monthly 
reports  identify  the  specific  user  of  the  research  results.  Of 
the  four  laboratories  we  reviewed,  two  issue  quarterly  reports, 
one  issues  monthly  reports,  and  one  does  not  produce  progress 
reports  of  any  kind. 

Another  concern  is  that  progress  reports  are  not  always 
distributed  to  the  specific  user  at  the  program  office  nor  is  the 
specific  user  otherwise  identified  in  the  report.  Without  a 
specific  user  being  identified  in  the  report,  it  is  difficult  to 
determine  what  research  is  currently  ongoing  to  fulfill  the  pro¬ 
gram  office  user's  needs.  For  example,  a  user  in  the  Office  of 
Air  Quality  Planning  and  Standards  told  us  that  he  saw  progress 
reports  on  a  grant  project  so  infrequently  that  he  had  lost 
track  of  the  project,  which  was  started  in  December  1976.  How¬ 
ever,  he  said  that  he  was  still  awaiting  the  final  report  and 
expected  it  to  be  marginally  useful.  Meanwhile,  the  ORD  project 
officer  had  accepted  the  grantee's  draft  of  the  final  report 
on  November  7,  1979,  as  fulfilling  the  grant  requirements.  The 
report  was  filed,  but  not  published  by  ORD.  The  project  officer 
told  us  that  he  wanted  to  reanalyze  and  reformat  some  of  the 
data  j->resented  in  the  draft  report  before  publishing  it,  but 
he  had  not  had  a  chance  to  do  so  as  of  March  1981. 
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We  found  that  informal  contacts  and  relationships  with  ORD 
staff  have  effectively  kept  many  intended  proyram  office  users 
aware  of  ORD’s  research  progress.  However,  as  notea  above,  this 
has  not  always  happened.  To  effectively  assure  that  users  are 
aware  of  the  research  being  performed  for  them,  specific  research 
users  need  to  be  routinely  identified  in  ORD's  progress  reports. 

ORD  project  tracking  system 

Another  information  system  that  could  be  used  to  monitor 
research  is  PTS.  This  system  was  developea  in  1977  to  be  used 
as  a  management  tool  and  information  source  for  coordinating 
research  projects.  This  system  contains  basic  data  on  4,b0u 
individual  ORD  research  projects,  over  2,500  of  which  are  ongoing. 
Under  the  PTS  system,  laboratories  submit  and  update  information 
for  all  ongoing  extramural  and  inhouse  research  projects  on  a 
quarterly  basis.  This  information  is  retained  for  5  years 
after  the  project's  completion.  A  PTS  project  printout 
includes  the  following  data: 

— Project  title. 

— Starting  and  completion  dates. 

— Inhouse  or  extramural  effort. 

— Prior,  current,  and  future  obligations. 

— Project  officer  and  telephone  number. 

— Investigators . 

— Program  title  and  legislative  mandate. 

— Results  to  date. 

— Project  status. 

— Project  abstract. 

Theoretically,  information  users  (in  and  outside  of  EPA) 
could  call  from  anywhere  in  the  country  and,  using  lowspeed 
computer  terminals,  gain  access  to  the  data  base.  However,  the 
PTS  has  not  been  operational  since  November  1980  due  to  adminis¬ 
trative  difficulties  and  project  constraints.  According  to 
ORD's  Office  of  Research  Management's  Deputy  Director,  EPA 
intends  to  reinstate  the  system  in  the  future. 


MORE  IMPROVEMENT  NEEDED 
IN  ASSURING  QUALITY  RESEARCH 


EPA  has  been  criticized  in  the  past  because  of  uneven 
research  quality.  Assuring  research  quality  is  a  difficult 
task,  but  generally  accepted  standards  of  scientific  quality 
include  peer  review,  publication  of  research  results  in 
scientific  journals,  and  a  sound  quality  assurance  program. 
Although  publishing  research  results  is  no  guarantee  of  high 
quality  research,  the  greater  the  extent  to  which  regulatory 
standards  are  based  on  published  data,  the  more  defensible  such 
standards  will  be  in  subsequent  litigation.  A  sound  quality 
assurance  program  is  regarded  as  the  first  and  necessary  criterion 
for  assuring  quality  research.  Responding  to  past  criticisms, 

EPA  established  a  program  that  provides  for  peer  review  ana 
publication  of  research  results  and  has  also  established  a  man¬ 
datory  quality  assurance  program. 

Although  significant  progress  has  been  made  regarding  peer 
review  and  publication  of  final  research  results,  we  believe  more 
needs  to  be  done  because 

— all  research  strategies  are  still  not  externally  reviewed 
and 

— peer  review  of  extramural  and  inhouse  research  proposals 
has  been  inconsistently  applied. 

The  mandatory  quality  assurance  program  was  not  in  place  long 
enough  to  gauge  its  effectiveness;  however,  the  planned  program 
seems  adequate. 

Significant  improvements  made  in 
peer  review  and  publication  of 
final  research  results 


EPA  has  made  significant  improvements  in  peer  reviewing  and 
publishing  final  research  results.  All  four  laboratories  we 
reviewed  provide  for  journal  publication  of  all  final  research 
results.  In  addition,  two  of  these  laboratories  provide  for 
external  peer  review  of  final  products,  and  the  remaining  two 
provide  for  external  peer  review  of  significant  or  politically 
sensitive  research. 

Although  important  progress  has  been  made  regarding  peer 
review  and  publication  of  final  research  results,  we  believe 
greater  use  of  external  peer  review  of  research  plans  and  pro¬ 
posals  is  still  needed.  A  1977  National  Academy  of  Science 
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report  1/  similarily  concluded  that  projects,  as  well  as  pro¬ 
posals,  must  be  reviewed  periodically  to  assure 

— scientific  and  technical  merit, 

— relevance  to  scientific  and  technical  program  goals,  and 

— relevance  of  programs  to  EPA's  mission. 

Because  the  credibility  of  research  performed  by  or  for  a 
regulatory  agency  is  sometimes  questioned,  the  Academy  stateu 
that  EPA  must  take  exceptional  measures  to  assure  that  its 
results  are  scientifically  valid. 

Strategy  documents  are 
not  externally  reviewed 


In  March  1979  EPA  strongly  encouraged  research  committees 
to  seek  external  peer  review  of  research  strategies  t->  gain  dif¬ 
ferent  perspectives  on  the  tasks  before  them.  EPA  also  agreed 
with  an  earlier  GAO  report  (CED-81-b,  Oct.  28,  198U)  that 
recommended  that  research  committees  be  required  to  obtain 
peer  review  of  their  multiyear  strategies.  We  found,  however, 
that  only  2  of  12  strategy  documents  for  1980-bl  were  subjected 
to  external  peer  review;  the  Science  Advisory  Board  reviewed 
the  1981  pesticides  document  and  the  1980  drinking  water  document 
was  reviewed  by  the  National  Drinking  Water  Advisory  Council. 

In  January  1981  the  Chairman  of  the  Science  Advisory  Board 
recommended  that  the  Board  work  with  and  provide  advice  on  the 
preparation  of  EPA's  research  strategies.  We  believe  such  exter¬ 
nal  peer  review  would  enhance  EPA's  prospects  of  achieving  a 
quality  research  program. 

Peer  review  of  research  proposals 
inconsistently  applied 

Major  improvements  in  peer  review  procedures  have  been 
.achieved  regarding  research  grants  and  cooperative  agreements. 
However,  research  proposals  for  contracts  and  inhouse  projects 
are  not  always  peer  reviewed. 

In  1977  the  National  Academy  of  sciences  stated  that  pro¬ 
posals  should  be  reviewed  to  assure  that  the  research  plan  is 
well  formulated,  that  it  has  a  reasonable  chance  to  meet  objec¬ 
tives,  and  that  the  researchers  have  adequate  funds,  facilities, 
and  expertise  to  accomplish  work.  Reviews  also  minimize  the 
risk  of  duplicating  work  already  done  elsewhere  and  may  help 
to  correct  a  limited  or  biased  perspective. 


1/ "Research  and  Development  in  the  Environmental  Protection  Agency, 
—  National  Academy  of  Sciences,  1977. 


Additionally,  the  Academy  stated  that  although  EPA  policy 
requires  reviews  of  proposals  from  non-Federal  sources,  proposals 
from  inside  EPA  and  from  other  Federal  agencies  are  not  generally 
submitted  for  external  review.  It  concluded  that  the  scientific 
merit  and  credibility  of  EPA' s  scientific  program  would  be 
enhanced  by  submitting  all  proposals  for  research,  without  excep¬ 
tion,  to  peer  review  by  scientists  both  inside  and  outside  EPA. 

ORD  grants,  cooperative 


agreements,  and  contracts 


Extramural  EPA  research  is  conducted  primarily  through  three 
methods — grants,  cooperative  agreements,  and  contracts.  EPA 
requires  all  research  proposals  that  will  be  formalized  into 
grants  and  cooperative  agreements  to  have  external  peer  review. 

Of  the  four  laboratories  we  reviewed,  all  required  such  peer 
review.  However,  EPA  has  not  required  contract  proposals  to  be 
externally  reviewed. 

We  found  that  research  contract  proposal  peer  review  proce¬ 
dures  are  inconsistent  between  laboratories.  Two  laboratories- 
require  external  peer  review  only  for  projects  that  exceed  a  cer¬ 
tain  dollar  limitation  ($250,000  and  $100,000).  Another  labora¬ 
tory  provides  for  external  peer  review  only  upon  the  judgment  of 
the  project  officer.  The  remaining  laboratory  provides  for 
inhouse  review  only. 

To  guara  against  uneven  quality  research  results,  we  believe 
EPA  needs  to  require  all  laboratories  to  subject  significant  con¬ 
tract  proposals  (preferably  those  over  $100,000)  to  external  peer 
review. 

Inhouse  research  proposals 

Although  approximately  29  percent — $103  million — of  EPA’ s 
research  budget  is  devoted  to  inhouse  research,  these  research 
proposals  are  not  consistently  peer  reviewed.  Of  the  four 
laboratories  we  reviewed,  two  require  external  peer  review  of 
their  more  significant  projects — those  exceeding  $250,000  or 
two  staff  years  of  effort.  The  remaining  two,  however,  only 
require  internal  review  of  their  proposals. 

Because  inhouse  research  comprises  such  a  large  portion 
of  EPA's  research  budget,  these  projects  also  need  to  be  sub¬ 
jected  to  the  rigors  of  external  peer  review. 

EPA's  planned  quality 
assurance  program  seems  adequate 


EPA  has  conceded  that,  in  the  past,  there  has  been  a  high 
degree  of  fragmentation,  lack  of  coordination,  poorly  identi¬ 
fied  needs  and  resources,  and  duplication  of  efforts  in  its 
quality  assurance  procedures.  For  these  reasons  it  established 
a  mandatory  quality  assurance  program  in  June  1979.  All  EPA 
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regional  offices,  program  offices,  laboratories,  and  those 
monitoring  and  measuring  efforts  supported  through  contracts 
or  other  formalized  agreements  are  required  to  participate  in 
the  program. 

The  primary  goal  of  the  program  is  to  ensure  that  all 
environmentally  related  measurements  supported  by  EPA  result  in 
data  of  known  quality.  To  meet  this  goal,  the  program  provides 
for  establishing  and  using  a  reliable  monitoring  and  measure¬ 
ment  system  to  obtain  quality  data  to  meet  EPA's  planned  needs. 

To  implement  this  policy,  each  EPA  laboratory,  program 
office,  and  regional  office  was  required  to  prepare  a  quality 
assurance  plan  covering  all  intramural  and  extramural  monitor¬ 
ing  and  measurement  activities.  These  plans  include  the  overall 
policies,  organization,  objectives,  and  functional  responsibili¬ 
ties  designed  to  achieve  data  quality  goals. 

In  addition,  quality  assurance  project  plans  are  required 
for  each  specific  research  project.  These  plans  describe  in 
more  specific  terms  the  quality  assurance  procedures  for  each 
project.  They  include  such  data  as  sampling  and  calibration 
procedures  and  internal  quality  control  checks. 

ORD's  quality  assurance  management  staff  is  primarily 
responsible  for  managing  this  program.  Among  other  things, 
it 

— reviews  and  approves  quality  assurance  plans  and 

— conducts  periodic  reviews  of  EPA's  quality  assurance  pro¬ 
grams  to  determine  if  deficiencies  exist  and  recommends 
corrective  actions . 

As  of  May  1,  1981,  27  of  the  44  required  quality  assurance 
plans  have  been  approved  by  the  quality  assurance  management 
staff.  The  remainder  are  still  in  draft  form.  Also,  as  of 
May  1,  1981,  the  first  series  of  quality  assurance  program 
audits  were  still  being  finalized. 

Because  EPA's  quality  assurance  program  is  still  being 
implemented,  we  believe  it  is  too  early  to  judge  its  effective¬ 
ness.  However,  based  on  our  review  of  the  program,  it  appears 
to  adequately  address  the  major  past  criticism  of  EPA's  quality 
controls . 

CONCLUSIONS 

Research  committess  have  the  responsibility  to  monitor 
research  progress  against  approved  strategies  and  implementation 
plans  and  suggest  corrections  for  significant  deviations.  Addi¬ 
tionally,  program  office  users  of  research  results  also  need  to 
be  kept  apprised  of  the  research  progress  and  have  knowledge  of 
any  significant  modifications  in  the  conduct  of  planned  research. 
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Although  some  research  committees  intend  to  formally  monitor 
the  progress  of  planned  research,  none  of  the  committees  we 
reviewed  have  done  this  to  date,  some  committees  we  reviewed 
intend  to  monitor  research  progress  in  the  future  using  existing 
ORD  information  systems.  However,  we  found  these  existing  systems 
are  either  inadequate  as  effective  mechanisms  for  monitoring 
research  progress  or  were  inoperative. 

Although  significant  progress  has  been  made  regarding  peer 
review  and  publication  of  final  research  results,  we  believe  more 
needs  to  be  done.  Specifically,  greater  use  of  external  peer 
review  of  research  strategies  and  extramural  and  inhouse  research 
proposals  are  needed. 

EPA’s  new  mandatory  quality  assurance  program  appears  to 
adequately  address  past  criticisms,  but  it  is  still  too  early 
to  gauge  its  effectiveness. 

RECOMMENDATIONS 

We  recommend  that  the  Administrator  of  EPA: 

— Require  research  committees  to  monitor  research  progress 
against  strategies  and  implementation  plans  and  ensure 
that  research  users  are  kept  apprised  of  research  progress. 

— Require  that  the  technical  information  plan  include  all 
ongoing  research  outputs  agreed  to  via  the  research  com¬ 
mittee  planning  process,  with  the  names  of  specific 
research  users  identified. 

— Require  ORD  progress  reports  to  identify  specific  research 
users  for  all  program-related  projects  and  require  these 
reports  to  be  distributed  to  the  specific  users. 

— Reinstate  the  Project  Tracking  System  to  include  the  names 
of  specific  users  for  all  program-related  projects. 

— Require  external  peer  review  of  all  research  strategies 
and  ensure  consistent  peer  review  of  extramural  and  inhouse 
research  proposals. 

ORD  COMMENTS  AND  OUR  EVALUATION 

ORD  officials  agreed  with  our  conclusions  and  recommenda¬ 
tions.  They  said  that  they  agree  monitoring  is  necessary  and 
that  research  committees  should  be  encouraged  to  monitor  research, 
but  felt  that  this  monitoring  should  consist  of  only  reviewing 
strategic  research  plans.  They  believe  monitoring  specific  pro¬ 
jects  should  be  the  principal  responsibility  of  ORD  managers. 

These  officials  did  agree  that  specific  users  could  be  identified 
on  EPA' s  various  project  monitoring  reports  but  would  prefer  that 
these  reports  be  distributed  to  specific  users  only  on  a  request 
basis.  We  continue  to  believe  that  research  committees  have  a 
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responsibility  to  monitor  research  progress  and  suggest  correc¬ 
tions  for  significant  deviations,  not  only  against  approved 
strategies  but  also  against  implementation  plans.  This  is  an 
integral  step  in  assuring  useful  research  results.  Similarly, 
we  believe  EPA  needs  to  distribute  its  project  monitoring  reports 
to  specific  users  of  research  results,  because  individual  users 
may  not  always  be  aware  that  a  research  project  is  underway  to 
fulfill  his  or  her  needs. 
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